Apparatus and methods for rate-modulated plasticity in a neuron network

ABSTRACT

Apparatus and methods for activity based plasticity in a spiking neuron network adapted to process sensory input. In one approach, the plasticity mechanism of a connection may comprise a causal potentiation portion and an anti-causal portion. The anti-causal portion, corresponding to the input into a neuron occurring after the neuron response, may be configured based on the prior activity of the neuron. When the neuron is in low activity state, the connection, when active, may be potentiated by a base amount. When the neuron activity increases due to another input, the efficacy of the connection, if active, may be reduced proportionally to the neuron activity. Such functionality may enable the network to maintain strong, albeit inactive, connections available for use for extended intervals.

PRIORITY AND CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a divisional and claims priority to co-pending U.S.patent application Ser. No. 13/774,934, entitled “APPARATUS AND METHODSFOR RATE-MODULATED PLASTICITY IN A SPIKING NEURON NETWORK”, filed onFeb. 22, 2013, the foregoing being incorporated herein by reference inits entirety. This application is related to co-owned U.S. patentapplication Ser. No. 13/152,119, entitled “SENSORY INPUT PROCESSINGAPPARATUS AND METHODS”, filed on Jun. 2, 2011, co-owned and co-pendingU.S. patent application Ser. No. 13/465,924, entitled “SPIKING NEURALNETWORK FEEDBACK APPARATUS AND METHODS”, filed May 7, 2012, co-owned andco-pending U.S. patent application Ser. No. 13/465,903 entitled “SENSORYINPUT PROCESSING APPARATUS IN A SPIKING NEURAL NETWORK”, filed May 7,2012, co-owned U.S. patent application Ser. No. 13/465,918, entitled“SPIKING NEURAL NETWORK OBJECT RECOGNITION APPARATUS AND METHODS”, filedMay 7, 2012, co-owned U.S. patent application Ser. No. 13/488,106,entitled “SPIKING NEURON NETWORK APPARATUS AND METHODS”, filed Jun. 4,2012, co-owned U.S. patent application Ser. No. 13/488,144, entitled“SPIKING NEURON NETWORK APPARATUS AND METHODS”, filed Jun. 4, 2012,co-owned U.S. patent application Ser. No. 13/541,531, entitled“CONDITIONAL PLASTICITY SPIKING NEURON NETWORK APPARATUS AND METHODS”,filed Jul. 3, 2012, U.S. patent application Ser. No. 13/548,071,entitled “SPIKING NEURON NETWORK SENSORY PROCESSING APPARATUS ANDMETHODS”, filed Jul. 12, 2012, co-owned U.S. patent application Ser. No.13/660,923, entitled “ADAPTIVE PLASTICITY APPARATUS AND METHODS FORSPIKING NEURON NETWORK”, Oct. 25, 2012, co-owned U.S. patent applicationSer. No. 13/660,967, entitled “APPARATUS AND METHODS FOR ACTIVITY-BASEDPLASTICITY IN A SPIKING NEURON NETWORK”, filed Oct. 25, 2012, co-ownedU.S. patent application Ser. No. 13/660,982, entitled “SPIKING NEURONSENSORY PROCESSING APPARATUS AND METHODS FOR SALIENCY DETECTION”, filedOct. 25, 2012, co-owned U.S. patent application Ser. No. 13/660,945,entitled “MODULATED PLASTICITY APPARATUS AND METHODS FOR SPIKING NEURONNETWORKS”, filed Oct. 25, 2012, co-owned U.S. patent application Ser.No. 13/691,554, entitled “RATE STABILIZATION THROUGH PLASTICITY INSPIKING NEURON NETWORK”, filed Nov. 30, 2012, and co-owned U.S. patentapplication Ser. No. 13/763,005, entitled “SPIKING NETWORK APPARATUS ANDMETHOD WITH BIMODAL SPIKE-TIMING DEPENDENT PLASTICITY”, filed Feb. 8,2013, each of the foregoing incorporated herein by reference in itsentirety.

COPYRIGHT

A portion of the disclosure of this patent document contains materialthat is subject to copyright protection. The copyright owner has noobjection to the facsimile reproduction by anyone of the patent documentor the patent disclosure, as it appears in the Patent and TrademarkOffice patent files or records, but otherwise reserves all copyrightrights whatsoever.

BACKGROUND

1. Technological Field

The present disclosure relates generally to artificial neural networks,and more particularly in one exemplary aspect to computer apparatus andmethods for plasticity implementation in a pulse-code neural network.

2. Description of Related Art

Artificial spiking neural networks are frequently used to gain anunderstanding of biological neural networks, and for solving artificialintelligence problems. These networks typically employ a pulse-codedmechanism, which encodes information using timing of the pulses. Suchpulses (also referred to as “spikes” or ‘impulses’) are short-lasting(typically on the order of 1-2 ms) discrete temporal events. Severalexemplary embodiments of such encoding are described in commonly ownedand co-pending U.S. patent application Ser. No. 13/152,084 entitledAPPARATUS AND METHODS FOR PULSE-CODE INVARIANT OBJECT RECOGNITION”,filed Jun. 2, 2011, the foregoing being incorporated herein by referencein its entirety, and U.S. patent application Ser. No. 13/152,119, Jun.2, 2011, entitled “SENSORY INPUT PROCESSING APPARATUS AND METHODS”,incorporated supra.

Typically, artificial spiking neural networks, such as the exemplarynetwork described in owned U.S. patent application Ser. No. 13/541,531,entitled “CONDITIONAL PLASTICITY SPIKING NEURON NETWORK APPARATUS ANDMETHODS”, incorporated supra, may comprise a plurality of units (ornodes), which can be thought of as corresponding to neurons in abiological neural network. Any given unit may be connected to many otherunits via connections, also referred to as communications channels,and/or synaptic connections. The units providing inputs to any givenunit are commonly referred to as the pre-synaptic units, while the unitsreceiving the inputs are referred to as the post-synaptic units.

Individual ones of the unit-to-unit connections may be assigned, interalia, a connection efficacy, which in general may refer to a magnitudeand/or probability of input spike influence on unit output response(i.e., output spike generation/firing). The efficacy may comprise, forexample a parameter (e.g., synaptic weight) by which one or more statevariables of post-synaptic unit are changed. The efficacy may comprise alatency parameter by characterizing propagation delay from apre-synaptic unit to a post-synaptic unit. In some implementations,greater efficacy may correspond to a shorter latency.

Some existing implementations of temporal learning (e.g., slow featureanalysis) by spiking neural networks via spike timing dependentplasticity and/or increased excitability may develop diminishedresponsiveness (‘forget’) features that did not appear for an extendedperiod of time (e.g., 10 minutes or longer for a 25 frames per second(fps) visual stimulus input).

Previously strong but presently inactive input synapses may becomedepressed based on the activity of the post synaptic neuron. Thisconfiguration may lead (especially in multi-layer processing networks)to unstable input synaptic sets and/or receptive fields.

Accordingly, there is a salient need for improved network operationcapable of, inter alia, responding efficiently to stimuli that mayappear at long intervals between one another.

SUMMARY OF THE DISCLOSURE

The present disclosure satisfies the foregoing needs by providing, interalia, apparatus and methods for implementing activity based plasticityin spiking neuron networks that is capable of, inter alia, respondingefficiently to infrequently appearing.

In a first aspect, a non-transitory computerized spiking neuronapparatus comprising a plurality of computer-readable instructions isdisclosed. In an implementation, when executed, the instructions areconfigured to, based on a response by the neuron, (i) increase anefficacy of a connection configured to provide input to the neuron priorto the response, and (ii) adjust the efficacy of a connection configuredto provide input to the neuron subsequent to the response.

In some implementations, the adjustment of the efficacy is determinedbased on a rate of the response.

In a second aspect, a sensory processing spiking neuron networkapparatus is disclosed. In one exemplary implementation, the apparatusincludes a connection configured to provide an input to the neuronconfigured to generate a response based on the input.

In various implementations, the connection is further configured to be:(i) potentiated when the input is within an interval from the response,and (ii) adjusted when the input occurs subsequent to the response, theadjustment being determined based on activity of the neuron prior to theresponse.

In a third aspect, a non-transitory computer-readable storage apparatushaving instructions embodied thereon is disclosed. In oneimplementation, when executed, the instructions are configured to, interalia, update a connection configured to provide stimulus to anartificial spiking neuron.

In one or more implementations, the update is configured to: (i)potentiate the connection if the stimulus precedes a response generatedby the neuron, and (ii) if the response precedes the stimulus: (a)potentiate the connection if neuron activity is below a threshold level,and (b) depress the connection if the neuron activity is above thethreshold level.

In a fourth aspect, a method of managing a connection in a spikingneuron network based on at least one signal from a neuron is disclosed.In an exemplary implementation, the method includes: (i) receiving atleast one input via the connection, (ii) sending the at least one signalat a time proximate to the received at least one input, (iii) if the atleast one signal is sent prior to the reception of the input, demotingthe connection, and (iv) if the at least one signal is sent after thereception of the input, promoting the connection.

In a fifth aspect, a discreet apparatus configured to manage response tostimuli in a neuron network is disclosed. In various implementations,the apparatus comprises a network element configured to temporallycorrelate feedback.

In a sixth aspect, a method of sensory processing via one or more ratemodulated mechanisms is disclosed.

Further features of the present disclosure, its nature and variousadvantages will be more apparent from the accompanying drawings and thefollowing detailed description.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a graphical illustration depicting an artificial spikingneural network configured for implementing activity modulatedplasticity, according to one or more implementations.

FIG. 2A is a plot illustrating connection potentiation via an activitymodulated plasticity mechanism, according to one or moreimplementations.

FIG. 2B is a plot illustrating connection depression via an activitymodulated plasticity mechanism, according to one or moreimplementations.

FIG. 2C is a plot illustrating adjusting efficacy a network connectionconfigure to provide, inter alia, input via an activity modulatedplasticity mechanism, according to one or more implementations.

FIG. 3A is a plot depicting activity modulated plasticity comprisingcausal potentiation, according to one or more implementations.

FIG. 3B is a plot depicting activity modulated plasticity comprisingconstant causal potentiation, according to one or more implementations.

FIG. 3C is a plot depicting activity modulated plasticity comprisingexponential causal potentiation, according to one or moreimplementations.

FIG. 3D is a plot depicting a resource depletion mechanism configuredfor activity modulated plasticity, according to one or moreimplementations.

FIG. 4A is a plot depicting activity modulated plasticity comprisingcausal and anti-causal potentiation and a multi-branch anti-casual rule,according to one or more implementations.

FIG. 4B is a plot depicting activity modulated plasticity comprisingcausal and anti-causal potentiation, according to one or moreimplementations.

FIG. 4C is a plot depicting activity modulated plasticity comprisingcausal and anti-causal potentiation, configured using a continuousfunction, according to one or more implementations.

FIG. 4D is a plot depicting activity modulated plasticity comprisingcausal and anti-causal potentiation, configured using a multi-branchdependency, according to one or more implementations.

FIG. 5 is a logical flow diagram illustrating a generalized method ofactivity-dependent plasticity, in accordance with one or moreimplementations.

FIG. 6 is a logical flow diagram illustrating a method of activitydependent plasticity comprising causal potentiation for use with thenetwork of FIG. 1, in accordance with one or more implementations.

FIG. 7 is a logical flow diagram illustrating a method of activitydependent plasticity comprising causal and anti-causal potentiation foruse with the network of FIG. 1, in accordance with one or moreimplementations.

FIG. 8 is a logical flow diagram illustrating a method efficacyadjustment determination of activity dependent plasticity comprisingcausal potentiation for use with the network of FIG. 1, in accordancewith one or more implementations.

FIG. 9 is a logical flow diagram illustrating a method efficacyadjustment based on response rate, in accordance with one or moreimplementations.

FIG. 10 is a block diagram illustrating a sensory processing apparatuscomprising adaptive plasticity mechanism in accordance with one or moreimplementations.

FIG. 11A is a block diagram illustrating a computerized systemconfigured to, inter alia, provide an adaptive plasticity mechanism in aspiking network, in accordance with one or more implementations.

FIG. 11B is a block diagram illustrating a neuromorphic computerizedsystem configured to operate in accordance with, inter cilia, anadaptive plasticity mechanism in a spiking network.

FIG. 11C is a block diagram illustrating a hierarchical neuromorphiccomputerized system architecture configured to operate in accordancewith, inter alia, an adaptive plasticity mechanism in a spiking network.

FIG. 11D is a block diagram illustrating cell-type neuromorphiccomputerized system architecture configured to operate in accordancewith, inter alia, an adaptive plasticity mechanism in a spiking network.

All Figures disclosed herein are © Copyright 2014 Brain Corporation. Allrights reserved.

DETAILED DESCRIPTION

Exemplary embodiments and implementations of the various aspects of thepresent disclosure will now be described in detail with reference to thedrawings, which are provided as illustrative examples so as to enablethose skilled in the art to practice the disclosure. Notably, thefigures and examples below are not meant to limit the scope of thepresent disclosure to a single embodiment or implementation, but otherembodiments and implementations are possible by way of interchange of orcombination with some or all of the described or illustrated elements.Wherever convenient, the same reference numbers will be used throughoutthe drawings to refer to same or like parts.

Where certain elements of these embodiments or implementations can bepartially or fully implemented using known components, only thoseportions of such known components that are necessary for anunderstanding of the present disclosure will be described, and detaileddescriptions of other portions of such known components will be omittedso as not to obscure the innovation.

In the present specification, an embodiment or implementations showing asingular component should not be considered limiting; rather, thedisclosure is intended to encompass other embodiments or implementationsincluding a plurality of the same component, and vice-versa, unlessexplicitly stated otherwise herein.

Further, the present disclosure encompasses present and future knownequivalents to the components referred to herein by way of illustration.

As used herein, the term “bus” is meant generally to denote all types ofinterconnection or communication architecture that is used to access thesynaptic and neuron memory. The “bus” could be optical, wireless,infrared or another type of communication medium. The exact topology ofthe bus could be for example standard “bus”, hierarchical bus,network-on-chip, address-event-representation (AER) connection, or othertype of communication topology used for accessing, e.g., differentmemories in pulse-based system.

As used herein, the terms “computer”, “computing device”, and“computerized device”, include, but are not limited to, personalcomputers (PCs) and minicomputers, whether desktop, laptop, orotherwise, mainframe computers, workstations, servers, personal digitalassistants (PDAs), handheld computers, embedded computers, programmablelogic device, personal communicators, tablet or “phablet” computers,portable navigation aids, J2ME equipped devices, cellular telephones,smart phones, personal integrated communication or entertainmentdevices, or literally any other device capable of executing a set ofinstructions and processing an incoming data signal.

As used herein, the term “computer program” or “software” is meant toinclude any sequence or human or machine cognizable steps which performa function. Such program may be rendered in virtually any programminglanguage or environment including, for example, C/C++, C#, Fortran,COBOL, MATLAB™, PASCAL, Python, assembly language, markup languages(e.g., HTML, SGML, XML, VoXML), and the like, as well as object-orientedenvironments such as the Common Object Request Broker Architecture(CORBA), Java™ (including J2ME, Java Beans), Binary Runtime Environment(e.g., BREW), and other languages.

As used herein, the terms “connection”, “link”, “synaptic channel”,“transmission channel”, “delay line”, are meant generally to denote acausal link between any two or more entities (whether physical orlogical/virtual), which enables information exchange between theentities.

As used herein, the term “memory” includes any type of integratedcircuit or other storage device adapted for storing digital dataincluding, without limitation, ROM. PROM, EEPROM, DRAM, Mobile DRAM,SDRAM, DDR/2 SDRAM, EDO/FPMS, RLDRAM, SRAM, “flash” memory (e.g.,NAND/NOR), memristor memory, and PSRAM.

As used herein, the terms “processor”, “microprocessor” and “digitalprocessor” are meant generally to include all types of digitalprocessing devices including, without limitation, digital signalprocessors (DSPs), reduced instruction set computers (RISC),general-purpose (CISC) processors, microprocessors, gate arrays (e.g.,field programmable gate arrays (FPGAs)), PLDs, reconfigurable computerfabrics (RCFs), array processors, secure microprocessors, andapplication-specific integrated circuits (ASICs). Such digitalprocessors may be contained on a single unitary IC die, or distributedacross multiple components.

As used herein, the term “network interface” refers to any signal, data,or software interface with a component, network or process including,without limitation, those of the FireWire (e.g., FW400, FW800, etc.),USB (e.g., USB2), Ethernet (e.g., 10/100, 10/100/1000 (GigabitEthernet), 10-Gig-E, etc.), MoCA, Coaxsys (e.g., TVnet™), radiofrequency tuner (e.g., in-band or OOB, cable modem, etc.), Wi-Fi(802.11), WiMAX (802.16), PAN (e.g., 802.15), cellular (e.g., 3G,LTE/LTE-A/TD-LTE, GSM, etc.) or IrDA families.

As used herein, the terms “pulse”, “spike”, “burst of spikes”, and“pulse train” are meant generally to refer to, without limitation, anytype of a pulsed signal, e.g., a rapid change in some characteristic ofa signal, e.g., amplitude, intensity, phase or frequency, from abaseline value to a higher or lower value, followed by a rapid return tothe baseline value and may refer to any of a single spike, a burst ofspikes, an electronic pulse, a pulse in voltage, a pulse in electricalcurrent, a software representation of a pulse and/or burst of pulses, asoftware message representing a discrete pulsed event, and any otherpulse or pulse type associated with a discrete information transmissionsystem or mechanism.

As used herein, the term “receptive field” is used to describe sets ofweighted inputs from filtered input elements, where the weights may beadjusted.

As used herein, the term refers to, without limitation, any of thevariants of IEEE-Std. 802.11 or related standards including 802.11a/b/g/n/s/v and 802.11-2012.

As used herein, the term “wireless” means any wireless signal, datacommunication, or other interface including without limitation Wi-Fi,Bluetooth, 3G (3GPP/3GPP2), HSDPA/HSUPA, TDMA, CDMA (e.g., IS-95A,WCDMA, etc.), FHSS, DSSS, GSM, PAN/802.15, WiMAX (802.16), 802.20,narrowband/FDMA, OFDM, PCS/DCS, LTE/LTE-A/TD-LTE, analog cellular, CDPD,RFID or NFC (e.g., EPC Global Gen. 2, ISO 14443, ISO 18000-3), satellitesystems, millimeter wave or microwave systems, acoustic, and infrared(e.g., IrDA).

The present disclosure provides, in one salient aspect, apparatus andmethods for implementing activity-based plasticity mechanisms in spikingneuron networks configured to, inter alia, to improve network responseto input stimuli comprising a variety of features.

In some implementations, a neuron may respond to different aspects ofthe input that may be separated by period of inactivity (e.g., a pausein the input). By way of non-limiting illustration, a neuron may respondto an aspect (e.g., a color and/or model) of a car in the video input.An object such as the car may initially appear in the input, may not besubsequently present in the input for several seconds. The plasticitymethodology described herein may enable maintaining strong, but inactiveconnections of a neuron, for extended periods of time while providingfor neuron response selectivity.

Implementations of the foregoing functionality of the present disclosuremay be useful in a variety of applications, including for instance aprosthetic device, autonomous robotic apparatus, and otherelectromechanical devices requiring visual or other sensory dataprocessing functionality.

Detailed descriptions of the various embodiments and implementations ofthe apparatus and methods of the disclosure are now provided. Althoughcertain aspects of the disclosure can best be understood in the contextof the sensory (e.g., visual) information processing using pulse-codeneural networks, the disclosure is not so limited, and implementationsof the disclosure may also be used in a wide variety of otherapplications, including for instance in implementing connectionadaptation in pulse-code neural networks.

Implementations of the disclosure may be for example deployed in ahardware and/or software realization of a neuromorphic computer system.In one such implementation, a robotic system may include a processorembodied in an application specific integrated circuit, which can beadapted or configured for use in an embedded application (such as aprosthetic device).

FIG. 1 illustrates a spiking neuronal network 100, configured to processsensory input, in accordance with one or more implementations. Thesensory input (the input 104 in FIG. 1) may comprise, for example, anaudio signal, a stream of video frames, and/or other input. In someimplementations, the input stimulus 104 may comprise image framesreceived from an image sensor (such as a CCD or a CMOS sensor device).In one or more implementations, the input may comprise a pixel streamdownloaded from a file, such as a stream of two-dimensional matrices ofred green blue RGB values (e.g., refreshed at a 25 Hz or other suitableframe rate). It will be appreciated by those skilled in the art whengiven this disclosure that the above-referenced image parameters aremerely exemplary, and many other image representations (e.g., bitmap,luminance-chrominance (YUV, YCbCr), cyan-magenta-yellow and key (CMYK),grayscale, etc.) are equally applicable to and useful with the variousaspects of the present disclosure. Furthermore, data framescorresponding to other (non-visual) signal modalities such as sonograms,IR, radar or tomography images are equally compatible with theprocessing network of FIG. 1, or yet other configurations.

The network 100 of FIG. 1 may comprise one (or more layers) of neurons(units) (depicted by triangles 102 in FIG. 1). The units 102 of onelayer may receive sensory input 104 as feed-forward stimulus. Individualneurons 102 may be characterized by receptive fields. In one or moreimplementations of visual stimulus possessing, the receptive fields maydescribe sensitivity of a neuron to the stimulus as a function of alocation of an object/feature (associated with the stimulus) within aframe (e.g., a horizontal and/or a vertical coordinate). Receptivefields with various spatial sensitivity such as for example, Gaussian,box cart, triangular, raised cosine and/or other dependencies, may beutilized when encoding sensory input.

In some implementations, the spiking neuron networks may employ aninhibition mechanism in order to increase competition between neuronsand to produce a variety of receptive fields responding to individualobjects, such as those described in detail in U.S. patent applicationSer. No. 13/152,105, entitled “APPARATUS AND METHODS FOR TEMPORALLYPROXIMATE OBJECT RECOGNITION”, filed Jun. 2, 2011, which is incorporatedherein by reference in its entirety.

A wide variety of competition mechanisms may be applied in conjunctionwith the learning principles discussed herein. For example, oneapproach, commonly referred to as “winner take all” (WTA), may allow asingle detector (for example neuron 135 of FIG. 1) to report detectionof the specific object. This is achieved by, inter alia, sending of acontrol (inhibition) signal from the detector (the neuron node that isthe first to detect the object) to other detectors to delay and/orprevent detection signal generation by the other detectors.

Returning now to FIG. 1, a portion of the sensory input 104 may beprovided to individual neurons 102 via one or more feed-forwardconnections 106, 108 (e.g., connections 106_1, 106_2 providing oneportion of the input to the neurons 102_1, 102_2, respectively andconnections 108_1, 108_2 providing another portion of the input to theneurons 102_1, 102_2, respectively). Individual connections 106, 108 maybe characterized by connection efficacy, denoted by open circles 108 inFIG. 1. In some implementations, connection efficacy may be used todescribe a magnitude, and/or a probability of influence of pre-synapticspike (e.g., the input 104 in FIG. 1) on response generation by apost-synaptic neuron (e.g., 102 in FIG. 1). In one or moreimplementations, connection efficacy may comprise for example aparameter such as, synaptic weight, by which one or more state variablesof the post-synaptic neuron(s) may be changed.

Individual neurons 102 may be operated in accordance with a neurondynamic process that may be characterized by a neuronal state parameter.In some implementations, the state parameter may comprise neuronexcitability, neuron membrane potential, and/or one or more otherparameters. The dynamic process may comprise a stochastic process, suchas for example those described in co-owned and co-pending U.S. patentapplication Ser. No. 13/487,533, entitled “STOCHASTIC SPIKING NETWORKLEARNING APPARATUS AND METHODS” filed Jun. 4, 2012, incorporated hereinby reference in its entirety.

The neurons 102 may generate responses based on the input 104 deliveredvia the one or more connections 106, 108. In some implementations, theresponse may be generated in accordance with a deterministic spikeresponse model, such as for example described in U.S. patent applicationSer. No. 13/152,119, entitled “APPARATUS AND METHODS FOR TEMPORALLYPROXIMATE OBJECT RECOGNITION”, filed on Jun. 2, 2011, incorporatedsupra. The spike response process may comprise an adaptive thresholdadjustment, such as e.g., described in U.S. patent application Ser. No.13/623,820, entitled “APPARATUS AND METHODS FOR ENCODING OF SENSORY DATAUSING ARTIFICIAL SPIKING NEURONS”, filed on Sep. 20, 2012 incorporatedherein by reference in its entirety. The neurons 102 may be operable inaccordance with stochastic process, such as for example that thosedescribed in U.S. patent application Ser. No. 13/487,499, entitled“STOCHASTIC APPARATUS AND METHODS FOR IMPLEMENTING GENERALIZED LEARNINGRULES”, filed on Jun. 4, 2012, incorporated herein by reference in itsentirety. Response generation in a stochastic neuron may becharacterized by a probability of response generation.

The responses of the neurons 102 may be communicated via feed-forwardconnections 112, for subsequent processing by other parts of thenetwork. In some implementations, such processing may comprise detectionof higher-order features (e.g., corners), objects (e.g., rectangles),and/or other applications. Typically, the feed-forward stimulus 104 maycause an increase of a probability of the response by the neuron 102.

The network 100, may be operated in accordance with activity-dependentplasticity mechanism described in detail with respect to FIGS. 2A-4Cbelow.

FIGS. 2A-2C illustrate the use of plasticity rules of the disclosure(e.g., the rule 340 of FIG. 340 of FIG. 3B) for modifying efficacy of aconnection (e.g., the connection 106_1 in FIG. 1). In FIG. 2A, the trace260 depicts post-synaptic unit activity (e.g., 102_1 in FIG. 1); traces266, 270 depict pre-synaptic input activity (e.g., the input 106_1,108_1). Based on the spike input 268 in FIG. 2A, the post-synaptic unitmay respond with the spike output 262 at times t₁, t₂, t₃. Thin lines inFIG. 2A (e.g., 230) depict temporal relationship between thepre-synaptic input and the earliest post-synaptic response that mayfollow the input. Based on individual spikes within the input 268preceding the respective post-synaptic spikes 262, efficacy of theconnection associated with the activity 266 may be increased. Theefficacy increase (indicated by the upward steps shown in the trace 274at times t₁, t₂, t₃) may performed be in accordance with thepotentiating portion (e.g., the interval 350) of the post-synaptic ruleportion 344 of FIG. 3B, shown by the panel 276 in FIG. 2A. Responsive toabsence of activity on the input trace 266 is present, the efficacy ofthe connection may be maintained substantially unchanged for t>t₃. Theefficacy behavior for t>t₃ may be based on STDP rule 344 portioncorresponding to the time interval 352 in FIG. 3B.

Based on the spike input 272 via the other connection 270 in FIG. 2A,the post-synaptic unit may respond with the spike output 264 at timest₄, t₅, t₆, t₇. Based on individual spikes within the input 262preceding the post-synaptic spikes 264, efficacy of the connection 270may be increased. The efficacy increase (indicated by the upward stepsshown in the trace 278 at times t₁, t₂, t₃) may be performed inaccordance with the potentiating portion (e.g., the interval 350) of thepost-synaptic rule portion 344 of FIG. 3B. It is noteworthy that evenwhen the post-synaptic activity is present at times t₄, t₅, t₆, t₇,because the output 264 is due to the input 272, the efficacy of theconnection associated with the activity of the trace 260 is maintainedunchanged, in accordance with the rule 276.

Individual input connections may be operated in operated in accordancewith activity-dependent plasticity rules, such as exemplaryimplementations described with respect to FIGS. 3A-4D.

FIGS. 3A-3B present activity modulated plasticity rules comprisingcausal potentiation, according to one or more implementations. Thecurves 300, 340 depict change in connection efficacy Δθ) determined as afunction of a delay between the input and the response:Δt=t_(post)−t_(pre). The rule 300 comprises a casual portion 304 (e.g.,corresponding to the post-synaptic response being generated subsequentto the pre-synaptic input); and an anti-casual portion 302 (e.g.,corresponding to the pre-synaptic input being subsequent to thepost-synaptic response). The casual portion comprises a potentiatingwindow 310 where the efficacy being increased (potentiated) by a maximummagnitude 306. As used hereinafter, the potentiation and/or depressionmagnitudes are referenced to the magnitude 306 (also denoted as Amax).For clarity, Amax may be set to unity in some implementations. Thecausal portion may be characterized by a maximum time interval Δt_(max)between the input and the output (e.g., 312). This maximum interval maybe configured such that inputs that precede the output by a time inexcess of the interval 312 may not cause connection potentiation (e.g.,the efficacy is maintained unchanged as (Δθ=0, Δt>Δt_(max)) as shown inFIG. 3A. The efficacy change associated with the anti-causal portion 302may be determined based on activity of the post-synaptic neuron. Theefficacy change may comprise connection potentiation (shown by the curve302_1 in FIG. 3A) or depression (shown by the curves 302_2, 302_3 inFIG. 3A). The magnitude 308 of efficacy change may be determined inaccordance with one or more rules described in detail below.

In some implementations, the anti-causal efficacy adjustment Δθ may bedetermined based on a function of the firing rate R of the post-synapticneuron as follows:Δθ(Δt)=ƒ(R).  (Eqn. 1)In one or more implementations the relationship ƒ(R) may comprise alinear function of the rate RΔθ(Δt)=−αR+β.  (Eqn. 2)The slope in Eqn. 2 may be configured as follows:α=−10β.  (Eqn. 3)Accordingly, for R>0.1 the rule of Eqn. 2 produces connection depression(Δθ<0); and for R>0.1 the rule of Eqn. 2 produces connectionpotentiation (Δθ>0).

In one or more implementations the relationship ƒ(R) may comprise aquadratic function of the rate RΔθ(Δt)=−α1R ²+β1.  (Eqn. 4)In some implementations the relationship ƒ(R) may comprise anexponential, and or a log functionΔθ(Δt)=−exp(α2R)+β2,  (Eqn. 5)Δθ(Δt)=−log(α3(R+R ₀))+β3.  (Eqn. 6)

In some implementations, the rate R utilized in of Eqn. 1-Eqn. 6 maycomprise an average rate of response of the neuron (e.g., firing rate).The average rate may be determined using a running mean, an exponentialaverage, a median, and or another approach. The average rate may bedetermined over a time interval T preceding the response. The interval Tmay be selected form the range between 1 s and 2000 s.

In one or more implementations, the relationship Eqn. 1 may comprise oneor more logic operations (e.g., a comparison operation between that theaverage response rate R and a minimum rate corresponding to apotentiation and or a depression of the connection.

In some approaches, a function of the inter-spike interval (ISI)distribution may be utilized to characterize neuron activity. The ISIdistribution may be determined based on a probability of spikesoccurring at times t and Δt as follows:ISI(Δt)=p((spike|_(t=D))&(spike|_(t=Δt)))  (Eqn. 7)

In one or more implementations, neuron activity may be described usingan average ISI, determined as follows:E

ISI

=∫₀ ^(∞)ISI(s)ds.  (Eqn. 8)

In one or more implementations, neuron activity may be characterizedusing a median value t_(m) of the ISI, determined as:∫₀ ^(t) ^(m) ISI(d)ds=∫ _(t) _(m) ^(∞)ISI(s)ds.  (Eqn. 9)

In some implementations, neuron activity may be characterized using anentropy of the ISI, determined as:H=∫ ₀ ^(∞)ISI(s)log(ISI(s))ds.  (Eqn. 10)

FIG. 3B illustrates an activity modulated plasticity rule comprisingconstant causal potentiation, according to one or more implementations.The rule 340 may provide for a reduced computational load, associatedwith performing plasticity updates in the spiking neuron network (SNN).

The rule 340 comprises a casual portion 344 (e.g., corresponding to thepost-synaptic response being generated subsequent to the pre-synapticinput); and an anti-casual portion 342 (e.g., corresponding to thepre-synaptic input being subsequent to the post-synaptic response). Thecasual portion comprises a potentiating window 310, wherein the efficacyis increased (potentiated) by a magnitude 346. The causal portion may becharacterized by a maximum time interval Δt_(max) between the input andthe output (e.g., 350) configured such that inputs that precede theoutput by a time in excess of the interval 350 may not cause connectionpotentiation and/or depression. For example, the efficacy may be remainunchanged (Δθ=0) over the duration 352 for Δt>Δt_(max)) as shown in FIG.3B. In one or more implementations, over the interval 352 the efficacymay be maintained within a 99.9% of the efficacy value at time Δt_(max).It will be appreciated by those skilled in the art that the above levelof efficacy variation within the interval 352 is exemplary and othertolerance numbers may be utilized, in accordance with the specificimplementation, provided that the efficacy is not increased/decreasedsubstantially during the interval 352. In some implementations, theefficacy behavior during the interval Δt>Δt_(max) may be described asfollows:∫_(Δt) _(max) ^(∞)Δθ(s)ds≦0.1∫₀ ^(Δt) ^(max) Δθ(s)ds  (Eqn. 11)

The efficacy change associated with the anti-causal portion 342 may bedetermined based on activity of the post-synaptic neuron. In one or moreimplementations, the efficacy change may comprise connectionpotentiation or depression, configured in accordance with Eqn. 1-Eqn. 6,described above.

In one or more approaches to processing inputs refreshed at 40 msintervals (25 fps), the potentiating intervals 310, 350 of the rules300, 340, respectively, may be selected from the range between 20 ms and200 ms; the interval 312 may be configured to be between 200 ms and 1000ms; the potentiation magnitude 309 may comprise a fraction (between, forexample, 0.05 and 0.2) of the maximum potentiation magnitude Amax (306);and the depression magnitude 309 may be configured comparable to themaximum potentiation magnitude Amax (306).

FIG. 2B illustrates connection depression due to lack of post-synapticoutput in response to pre-synaptic input. In FIG. 2B, the traces 280,284 depict post-synaptic unit activity and pre-synaptic input,respectively. Based on high post-synaptic activity (the spikes 282) andabsence of corresponding input on the trace 280, the anti-causal portionof the STDP rule (e.g., 342 in FIG. 3B) may be configured to causesubstantial depression, as illustrated by the STDP panel 286 in FIG. 2B.Responsive to the activity 288 at times t₁, t₂, t₃, t₄, t₅ on the trace284, the efficacy of the respective connection may be reduced, asillustrated by downward steps of the trace 289 corresponding to timeinstances t₁, t₂, t₃, t₄. In some implementations, such as shown in FIG.2B, connection efficacy may not be reduced at time t₅ due to efficacydepletion control mechanism, described with respect to FIG. 3D, below.

FIG. 2C illustrates connection potentiation due to pre-synaptic input inabsence of post-synaptic activity. In FIG. 2C, the traces 290, 292depict post-synaptic unit activity and pre-synaptic input, respectively.The initially inactive pre-synaptic connection (as illustrated byabsence of spikes on the trace 292 prior to time t₁ in FIG. 2C) maybecome active and deliver spiking input 294 at times t₁, t₂, t₃, t₄.Based on low (and/or absent post-synaptic activity) prior to the input294, the anti-causal portion of the STDP rule for the connection 292(e.g., 342 in FIG. 3B) may be configured to cause moderate potentiation,as illustrated by the STDP panel 296 in FIG. 2C. When the connectionassociated with the activity trace 292 becomes active and deliversspiking input 294, the connection efficacy may be increased, asillustrated by increments to the trace 299 corresponding to timeinstances t₁, t₂, t₃, t₄, in FIG. 2C.

FIG. 3C illustrates activity modulated plasticity rule comprisingexponential causal potentiation, according to one or moreimplementations. The rule 340 may provide for a reduced computationalload, associated with performing plasticity updates in the SNN.

The rule 360 comprises a casual portion 364 (e.g., corresponding to thepost-synaptic response being generated subsequent to the pre-synapticinput); and an anti-casual portion 362 (e.g., corresponding to thepre-synaptic input being subsequent to the post-synaptic response). Thecasual portion 364 is configured to decay exponentially with a delaybetween the pre-synaptic input and the post-synaptic response. In one ormore implementations, the decay time scale 366 may comprise e-foldingduration (e.g., the duration where the magnitude is reduced by factor of1/exp(1)) that may be configured to be between 10 ms and 50 ms. Theanti-casual portion 362 may be configured to cause connection depression(characterized by the interval 368 in FIG. 3C) and connectionpotentiation based on a delay between the post-synaptic response and thesubsequent input. The efficacy change associated with the anti-causalportion 362 may be determined based on activity of the post-synapticneuron. In one or more implementations, the efficacy change may compriseconnection potentiation or depression, configured in accordance withEqn. 1-Eqn. 6, described above.

FIG. 3D illustrates efficacy depletion control mechanism useful, forexample, with the anti-causal portions of the activity based plasticityrules 302, 342, 362, 422 described with respect to FIGS. 3A-3C, andFIGS. 4A-4D, respectively. In one or more implementations, the efficacydepletion mechanism may be effectuated by using a resource trace C(t).The traces 380, 387, 390, 396 depict the connection resource C, theconnection efficacy θ, the pre-synaptic input and the post-synapticactivity, respectively.

Responsive to the pre-synaptic input (e.g., pulses 392 in FIG. 3D) attimes t₁, t₂, t₃, t₄, t₅ not followed by the post-synaptic output withina potentiating window (e.g., the window 350 in FIG. 3B) the connectionmay be depressed as illustrated by decrements of the trace 387 at timest₁, t₂, t₃, t₄. Efficacy decrease may be configured using the activitybased STDP modulation (e.g., anti-causal rule 302_3 of FIG. 3A).Individual efficacy decrements (e.g., 385, 387) may be accompanied bydecrements of the resource C (e.g., decrements 382 in FIG. 3D). Based ondepletion of the resource C below a threshold value C_(min), (e.g.,C_(min)=0 in FIG. 3D), the connection may not be depressed further, asindicated by absence of the efficacy 387 decrease at time t₅ in FIG. 3D.In some implementations, such as illustrated in FIG. 3D, the efficacyadjustment may be based on the resource level |Δθ|˜ƒ(C). As shown inFIG. 3D, efficacy decrease 385 is greater in magnitude than the decrease386 at time t₁. In some implementations, (not shown) the connectionefficacy adjustment may be independent of the resource value C, providedC>C_(min).

The resource may be replenished responsive to a post-synaptic responseis generated, as illustrated by the resource increase 384 at time t₆ ofthe post-synaptic response 398 in FIG. 3D. Based on the occurrence ofone or more subsequent pre-synaptic inputs on the connection 390 (e.g.,the spikes 394) the connection efficacy may be further decreased, asshown by the decreases 389 at times t₇, t₈ in FIG. 3D.

FIG. 4A illustrates activity modulated plasticity rule comprising causaland-anti-causal potentiation, according to one or more implementations.The rule 420 of FIG. 4A may comprise a potentiating portion 424configured to potentiate connections providing inputs that are proximate(e.g., within the time interval 430) from the post-synaptic response.The casual portion of the rule 420 may be configured to leave connectionefficacy unchanged, based on the inputs provided by the connectionpreceding the response by a time interval in excess of the range 432 inFIG. 4A.

The anti-casual portion of the rule 420 may comprise a depressionportion 422 configured in accordance with any applicable activity basedmethodologies described herein. The maximum depression magnitude 434 ofthe anti-causal rule 422 may be configured comparable to the maximumpotentiation magnitude 428. During network operation, the actualmagnitude 434 of the anti-causal rule 422 may be configured based on theneuron response rate such that the rule portion 422 may causepotentiation or depression. In applications of with processing inputsrefreshed at 40 ms intervals (25 fps), the duration 430 of thepotentiation portion may range from 20 ms to 400 ms, inclusive.

In some implementations, the plasticity rule comprising causal andanti-causal potentiation may comprise a potentiating portion (e.g.,associated with the window 430 in FIG. 4A) may be characterized byconstant magnitude (not shown).

FIG. 4B illustrates activity modulated plasticity comprising causal andanti-causal potentiation, according to one or more implementations. Therule 440 may comprise a casual portion 444 configured to potentiateconnections providing inputs that are proximate (e.g., within the timeinterval 449) prior to the post-synaptic response. Casual portion 444 ofthe rule 440 may be configured to leave connection efficacy unchanged,based on the inputs provided by the connection preceding the response bya time interval in excess of the range 449 in FIG. 4B.

The anti-casual portion of the rule 440 of FIG. 4B may comprise apotentiation portion 443 and a depression portion 442. The potentiationportion 443 may be configured to potentiate connections providing inputsthat are proximate (e.g., within the time interval 447 in FIG. 4B)subsequent to the post-synaptic response. The maximum potentiationmagnitude of the anti-causal rule portion 432 may be configured on theorder of the maximum potentiation magnitude 448. In one or moreimplementations of processing inputs refreshed at 40 ms intervals (25fps), duration of individual time windows 447, 449 may be configuredbetween 20 ms and 200 ms.

The anti-casual depression portion 442 may be configured in accordancewith any applicable activity based methodologies described herein. Themaximum depression magnitude 446 of the anti-causal rule 442 may beconfigured comparable to the maximum potentiation magnitude 448. Duringnetwork operation, the actual magnitude 446 of the anti-causal rule 442may be configured based on the neuron response rate so that the ruleportion 442 may cause potentiation or depression.

FIG. 4C illustrates activity modulated plasticity realization 450comprising causal 454 and anti-causal 452 rules, according to one ormore implementations. The anti-casual rule 452 may be configured using acontinuous dependency. The anti-casual rule 452 may comprise a portionconfigured in accordance with any applicable activity basedmethodologies described herein. During network operation, the actualmagnitude 456 of the anti-causal rule 452 may be configured based on theneuron response rate so that the rule portion 452 may cause potentiationor depression.

FIG. 4D illustrates activity modulated plasticity rule 460 comprisingcausal 464 and anti-causal 462 portions, according to one or moreimplementations. The anti-casual rule portion 462 may be configuredusing two or more branches, e.g., as illustrated in FIG. 4D. Theanti-casual rule 462 may comprise a portion configured in accordancewith any applicable activity based methodologies described herein.During network operation, the actual magnitude 466 of the anti-causalrule 462 may be configured based on the neuron response rate so that therule portion 452 may cause potentiation or depression.

FIGS. 5-9 illustrate exemplary methods of using the activity basedplasticity mechanism in the operation of spiking neuron networks. In oneor more implementations, the operations of methods 500, 600, 700, 800,900 of FIGS. 5-9, respectively, may be effectuated by a processingapparatus comprising a spiking neuron network such as the apparatus 1000of FIG. 10, described in detail below.

FIG. 5 illustrates an exemplary implementation of a generalized methodof using activity-dependent plasticity in a spiking neuron networkprocessing apparatus. At operation 502 of method 500, input may bereceived by a spiking neuron of the network. The input may compriseimage frames received from an image sensor (such as a CCD or a CMOSsensor device). In one or more implementations, the input may comprise apixel stream downloaded from a file, such as a stream of two-dimensionalmatrices of red green blue RGB values (e.g., refreshed at a 25 Hz orother suitable frame rate). It will be appreciated by those skilled inthe art when given this disclosure that the above-referenced imageparameters are merely exemplary, and many other image representations(e.g., bitmap, luminance-chrominance (YUV, YCbCr), cyan-magenta-yellowand key (CMYK), grayscale, etc.) may be utilized with the variousaspects of the present disclosure. Furthermore, data framescorresponding to other (non-visual) signal modalities such as sonograms,IR, radar or tomography images are equally compatible with theprocessing network of FIG. 1, or yet other configurations.

At step 504, a determination may be made as to whether the response hasbeen generated and a plasticity update is to be performed. In one ormore implementations, the update may be based for example on an externalevent (e.g., reinforcement signal); a timer event (e.g., for cyclicupdates); or a buffer overflow event (e.g., indicative of a memorybuffer, storing, e.g., pre-synaptic and/or post-synaptic spike history)being full or nearly full. The history may comprise time data ofpre-synaptic spikes stored in a synaptic memory buffer, such asdescribed for example in U.S. patent application Ser. No. 13/239,259,entitled “APPARATUS AND METHOD FOR PARTIAL EVALUATION OF SYNAPTICUPDATES BASED ON SYSTEM EVENTS”, filed on Sep. 21, 2011, incorporatedsupra.

When the update is to be performed, the method may proceed to step 608,where a post-synaptic update may be performed. In one or moreimplementations, the post-synaptic update may comprise e.g., one or morepost-synaptic rule portions 300, 320, 340 illustrated in FIGS. 3A-3C.

If it is determined that the update is to be performed, the method mayproceed to operation 506 where a determination may be made as to whetherthe input occurred prior to the response.

If the input has occurred prior to the response, the method may proceedto operation 508 where a causal plasticity rule may be applied. In oneor more implementations, the casual rule may be configured in accordancewith one or more rules 304, 344, 364, 424 described above with respectto FIGS. 3A-4D.

If the input has occurred subsequent to the response, the method mayproceed to operation 510 wherein an anti-causal plasticity rule may beapplied. In one or more implementations, the anti-casual rule may beconfigured in accordance with one or more rules 302, 342, 362, 422described above with respect to FIGS. 3A-4D.

FIG. 6 illustrates a method of activity dependent plasticity comprisingcausal potentiation (such as, e.g., rule 340 of FIG. 3B) for use withthe network of FIG. 1, in accordance with one or more implementations.

At operation 602 a determination may be made as to whether the inputoccurred prior to the response. In one or more implementations, time ofoccurrence of individual pre-synaptic and/or post-synaptic events (e.g.,262, 268, 264, 272 in FIG. 2A) may be utilized in order to make thedetermination of operation 602. In some implementations, a timedifference between a respective pre-synaptic and/or post-synaptic eventand a reference event (e.g., frame onset and/or timer alarm) may beutilized.

If the input has occurred prior to the response, the method may proceedto operation 604 where a determination may be made as to whether a timedifference between the response and the input is within a potentiatinginterval (e.g., 350 in FIG. 3B).

If the delay between the input the response is within the potentiatinginterval, the connection may be potentiated at step 608.

If the delay between the input the response is longer than thepotentiating interval, the connection efficacy may remain unchanged.

If the input has occurred subsequent to the response, the method mayproceed to operation 606 where the connection may be potentiated ordepressed in accordance with the activity based plasticity mechanism. Inone or more implementations, the plasticity rule may be configured basedon one or more of Eqn. 1-Eqn. 6, described above.

FIG. 7 illustrates an exemplary method of activity dependent plasticitycomprising causal and anti-causal potentiation (such as, e.g., rule 420of FIG. 4A) for use with the network of FIG. 1.

At operation 702 of method 700 a determination may be made as to whetherthe input has occurred outside a time interval (e.g., 430 in FIG. 4A)from the response.

If the input has occurred within the potentiating interval from theresponse, the method may proceed to operation 710 where connection maybe potentiated. In one or more implementations, the potentiation maycomprise applying the potentiating portion 424 of rule 420 of FIG. 4A.

When it is determined at operation 702 that the input has occurredoutside the potentiating window, the method may proceed to operation 704wherein a determination may be made as to whether the input occurredprior to the response.

If the input has occurred prior to the response, the method may proceedto step 708 wherein the connection efficacy may remain unchanged.

If the input has occurred subsequent to the response, the method mayproceed to operation 706 wherein the connection may be potentiated ordepressed in accordance with the activity based plasticity mechanism. Inone or more implementations, the plasticity rule may be configured basedon one or more of Eqn. 1-Eqn. 6, described above.

FIG. 8 illustrates a method efficacy adjustment determination for usewith activity dependent plasticity rules of FIGS. 3A-4D, in accordancewith one or more implementations.

At operation 802 of method 800 a determination may be made as to whetherthe neuron activity is below a potentiation threshold. In one or moreimplementations, the threshold may correspond to an average firing rateconfigured such that when the average rate is below the threshold theconnection is potentiated; and when the average rate is above thethreshold, the connection may be depressed. In one or moreimplementations the comparison of operation 802 may be implicitlyperformed by configuring parameters of the rules of Eqn. 1-Eqn. 6, e.g.,the relationship Eqn. 3.

If it is determined at operation 802 that the activity is below thethreshold (e.g., the neuron responds infrequently), the connectionadjustment may comprise an efficacy increase configured, for example,determined at operation 804 using one or more of Eqn. 1-Eqn. 6.

If it is determined at operation 802 that the activity is above thethreshold (e.g., the neuron responds infrequently), the connectionadjustment may comprise an efficacy decrease configured, for example, asdetermined at operation 808 using one or more of Eqn. 1-Eqn. 6.

At operation 808, connection efficacy may be updated (e.g., potentiatedof depressed) based on various determinations of the operations 804,806.

FIG. 9 illustrates an exemplary method of adjusting connection efficacybased on a neuron response rate.

At operation 902 of method 900, post-synaptic activity history of aneuron may be accessed. In some implementations, the history maycomprise time data of post-synaptic spikes stored in a neuron privateand/or shared memory. In some implementations, the history may comprisetime data of (i) pre-synaptic spikes stored in a synaptic memory buffer,such as those described for example in U.S. patent application Ser. No.13/239,259, entitled “APPARATUS AND METHOD FOR PARTIAL EVALUATION OFSYNAPTIC UPDATES BASED ON SYSTEM EVENTS”, filed on Sep. 21, 2011,incorporated supra; and (ii) delays between the pre-synaptic input andthe respective post-synaptic spikes (e.g., a time interval 217 betweentime t2 and t1 in FIG. 2A).

At operation 904, a rate of response may be determined based on thehistory of post-synaptic activity. In one or more implementations, theresponse rate may comprise an average rate, determined using a runningmean, an exponential average, a median of the activity history values,and or another approach. The average rate may be determined over a timeinterval T preceding the response. The interval T may be selected formthe range between 1 s and 2000 s. In one or more implementations, theaverage rate may be determined using an exponential running average.

At operation 906 an efficacy adjustment (corresponding to the rate ofresponse produced by operation 904 above) may be determined. In one ormore implementations, the efficacy adjustment may be determined usingone or more of Eqn. 1-Eqn. 6.

At operation 908 a determination may be made as to whether the rate Rproduced by operation 904 above exceeds a minimum rate (Rmin)corresponding to connection potentiation.

If it is determined that R>Rmin, the method may proceed to operation 912where connection may be potentiated (efficacy increased).

If it is determined that R<Rmin, the method may proceed to operation 916where connection may be depressed (efficacy decreased).

Various exemplary spiking network apparatus implementing one or more ofthe methods set forth herein (e.g., using the exemplary activitydependent plasticity mechanisms described above) are now described withrespect to FIGS. 10-11D.

One exemplary apparatus for processing of sensory information (e.g.,visual, audio, somatosensory) using a spiking neural network (includingone or more of the activity dependent plasticity mechanisms describedherein) is shown in FIG. 10. The illustrated processing apparatus 1000includes an input interface configured to receive an input sensorysignal 1010. In some implementations, this sensory input compriseselectromagnetic waves (e.g., visible light, IR, UV, and/or otherwavelength) entering an imaging sensor array (comprising RGCs, a chargecoupled device (CCD), CMOS device, or an active-pixel sensor (APS)). Theinput signal in this example is a sequence of images (image frames)received from a CCD or a CMOS camera via a receiver apparatus, ordownloaded from a file. Alternatively, the image may be atwo-dimensional matrix of RGB values refreshed at a 25 Hz frame rate. Itwill be appreciated by those skilled in the art that the above imageparameters and components are merely exemplary, and many other imagerepresentations (e.g., bitmap, CMYK, grayscale, and/or anotherrepresentation) and/or frame rates may be utilized with the presentdisclosure.

The apparatus 1000 may comprise an encoder 1020 configured to transform(encode) the input signal so as to form an encoded signal 1024. In onevariant, the encoded signal comprises a plurality of pulses (alsoreferred to as a group of pulses) configured to model neuron behavior.The encoded signal 1024 may be communicated from the encoder 1020 viamultiple connections (also referred to as transmission channels,communication channels, or synaptic connections) 1004 to one or moreneuronal nodes (also referred to as the detectors) 1002.

In the implementation of FIG. 10, different detectors of the samehierarchical layer are denoted by an “_n” designator, such that e.g.,the designator 1002_1 denotes the first detector of the layer 1002.Although only two detectors (1002_1, 1002_n) are shown in FIG. 10 forclarity, it is appreciated that the encoder can be coupled to any numberof detector nodes that is compatible with the detection apparatushardware and/or software resources. Furthermore, a single detector nodemay be coupled to any practical number of encoders.

In one implementation, each of the detectors 1002_1, 1002_n containlogic (which may be implemented as a software code, hardware logic, or acombination of thereof) configured to recognize a predetermined patternof pulses in the encoded signal 1004. To produce post-synaptic detectionsignals transmitted over communication channels 1008 variousimplementations may use, for example, any of the mechanisms described inU.S. patent application Ser. No. 12/869,573, filed Aug. 26, 2010 andentitled “SYSTEMS AND METHODS FOR INVARIANT PULSE LATENCY CODING”, U.S.patent application Ser. No. 12/869,583, filed Aug. 26, 2010, entitled“INVARIANT PULSE LATENCY CODING SYSTEMS AND METHODS”, U.S. patentapplication Ser. No. 13/117,048, filed May 26, 2011 and entitled“APPARATUS AND METHODS FOR POLYCHRONOUS ENCODING AND MULTIPLEXING INNEURONAL PROSTHETIC DEVICES”, U.S. patent application Ser. No.13/152,084, filed Jun. 2, 2011, entitled “APPARATUS AND METHODS FORPULSE-CODE INVARIANT OBJECT RECOGNITION”, each of which beingincorporated herein by reference in its entirety. In FIG. 10, thedesignators 1008_1, 1008_n denote output of the detectors 1002_1,1002_n, respectively.

In one implementation, the detection signals are delivered to a nextlayer of the detectors 1012 (comprising detectors 1012_1, 1012_m,1012_k) for recognition of complex object features and objects, similarto the exemplary configuration described in commonly owned andco-pending U.S. patent application Ser. No. 13/152,084, filed Jun. 2,2011, entitled “APPARATUS AND METHODS FOR PULSE-CODE INVARIANT OBJECTRECOGNITION”, incorporated herein by reference in its entirety. In thisconfiguration, each subsequent layer of detectors is configured toreceive signals from the previous detector layer, and to detect morecomplex features and objects (as compared to the features detected bythe preceding detector layer). For example, a bank of edge detectors isfollowed by a bank of bar detectors, followed by a bank of cornerdetectors and so on, thereby enabling alphabet recognition by theapparatus.

Individual detectors 1002 may output detection (post-synaptic) signalson communication channels 1008_1, 1008_n (with appropriate latency) thatmay propagate with different conduction delays to the detectors 1012.The detector cascade of the apparatus of FIG. 10 may contain anypractical number of detector nodes and detector banks determined, intercilia, by the software/hardware resources of the detection apparatus,and/or complexity of the objects being detected.

The sensory processing apparatus implementation illustrated in FIG. 10may further comprise lateral connections 1006. In some variants, theconnections 1006 are configured to communicate post-synaptic activityindications between neighboring neurons of the same hierarchy level, asillustrated by the connection 1006_1 in FIG. 10. The neighboring neuronmay comprise neurons having overlapping inputs (e.g., the inputs 1004_1,1004_n in FIG. 10), so that the neurons may compete in order to notlearn the same input features. In one or more implementations, theneighboring neurons may comprise spatially proximate neurons such asbeing disposed within a certain volume/area from one another on a3-dimensional (3D) and or two-dimensional (2D) space.

The apparatus 1000 may also comprise feedback connections 1014,configured to communicate context information from detectors within onehierarchy layer to previous layers, as illustrated by the feedbackconnections 1014_1 in FIG. 10. In some implementations, the feedbackconnection 1014_2 is configured to provide feedback to the encoder 1020,thereby facilitating sensory input encoding, as described in detail incommonly owned and co-pending U.S. patent application Ser. No.13/152,084, filed Jun. 2, 2011, entitled “APPARATUS AND METHODS FORPULSE-CODE INVARIANT OBJECT RECOGNITION”, incorporated supra.

One particular implementation of the computerized neuromorphicprocessing system, adapted for operating a computerized spiking network(and implementing the exemplary plasticity methodology described supra),is illustrated in FIG. 11A. The computerized system 1100 of FIG. 11Acomprises an input interface 1110, such as for example an image sensor,a computerized spiking retina, an audio array, a touch-sensitive inputdevice, and/or another JO device. The input interface 1110 is coupled tothe processing block (e.g., a single or multi-processor block) via theinput communication interface 1114. The system 1100 further comprises arandom access memory (RAM) 1108, configured to store neuronal states andconnection parameters (e.g., weights 108 in FIG. 1A), and to facilitatesynaptic updates. In some exemplary implementations, synaptic updatesare performed according to the description provided in, for example, inU.S. patent application Ser. No. 13/239,255 filed Sep. 21, 2011,entitled “APPARATUS AND METHODS FOR SYNAPTIC UPDATE IN A PULSE-CODEDNETWORK”, incorporated by reference supra.

In some implementations, the memory 1108 is coupled to the processor1102 via a direct connection (memory bus) 1116. The memory 1108 may alsobe coupled to the processor 1102 via a high-speed processor bus 1112).

The system 1100 may further comprise a nonvolatile storage device 1106,comprising, inter alia, computer readable instructions configured toimplement various aspects of spiking neuronal network operation (e.g.,sensory input encoding, connection plasticity, operation model ofneurons, and/or other processing functions). The nonvolatile storage1106 may be used for instance to store state information of the neuronsand connections when, for example, saving/loading network statesnapshot, or implementing context switching (e.g., saving currentnetwork configuration (comprising, inter alia, connection weights andupdate rules, neuronal states and learning rules, and/or otherparameters) for later use, and/or loading of a previously stored networkconfiguration.

In some implementations, the computerized apparatus 1100 may be coupledto one or more of an external processing device, a storage device, aninput device, and/or other devices via an I/O interface 1120. The I/Ointerface 1120 may include one or more of a computer I/O bus (PCI-E),wired (e.g., Ethernet) or wireless (e.g., Wi-Fi) network connection,and/or other I/O interfaces.

In some implementations, the input/output (I/O) interface may comprise aspeech input (e.g., a microphone) and a speech recognition moduleconfigured to receive and recognize user commands.

It will be appreciated by those skilled in the arts that variousprocessing devices may be used with computerized system 1100, includingbut not limited to, a single core/multicore CPU, DSP, FPGA, GPU, ASIC,combinations thereof, and/or other processors. Various user input/outputinterfaces may be similarly applicable to implementations of theinvention including, for example, an LCD/LED monitor, touch-screen inputand display device, speech input device, stylus, light pen, trackball,and/or other devices.

Referring now to FIG. 11B, one implementation of neuromorphiccomputerized system configured to implement classification mechanismusing a spiking network is described in detail. The neuromorphicprocessing system 1130 of FIG. 11B may comprise a plurality ofprocessing blocks (micro-blocks) 1140. Individual micro cores maycomprise a computing logic core 1132 and a memory block 1134. The logiccore 1132 may be configured to implement various aspects of neuronalnode operation, such as the node model, and synaptic update rules and/orother tasks relevant to network operation. The memory block may beconfigured to store, inter cilia, neuronal state variables andconnection parameters (e.g., weights, delays, I/O mapping) ofconnections 1138.

The micro-blocks 1140 may be interconnected with one another usingconnections 1138 and routers 1136. As it is appreciated by those skilledin the arts, the connection layout in FIG. 11B is exemplary, and manyother connection implementations (e.g., one to all, all to all, and/orother maps) are compatible with the disclosure.

The neuromorphic apparatus 1130 may be configured to receive input(e.g., visual input) via the interface 1142. In one or moreimplementations, applicable for example to interfacing with computerizedspiking retina, or image array, the apparatus 1130 may provide feedbackinformation via the interface 1142 to facilitate encoding of the inputsignal.

The neuromorphic apparatus 1130 may be configured to provide output viathe interface 1144. Examples of such output may include one or more ofan indication of recognized object or a feature, a motor command (e.g.,to zoom/pan the image array), and/or other outputs.

The apparatus 1130, in one or more implementations, may interface toexternal fast response memory (e.g., RAM) via high bandwidth memoryinterface 1148, thereby enabling storage of intermediate networkoperational parameters. Examples of intermediate network operationalparameters may include one or more of spike timing, neuron state, and/orother parameters. The apparatus 1130 may interface to external memoryvia lower bandwidth memory interface 1146 to facilitate one or more ofprogram loading, operational mode changes, retargeting, and/or otheroperations. Network node and connection information for a current taskmay be saved for future use and flushed. Previously stored networkconfiguration may be loaded in place of the network node and connectioninformation for the current task, as described for example in co-pendingand co-owned U.S. patent application Ser. No. 13/487,576 entitled“DYNAMICALLY RECONFIGURABLE STOCHASTIC LEARNING APPARATUS AND METHODS”,filed Jun. 4, 2012, incorporated herein by reference in its entirety.External memory may include one or more of a Flash drive, a magneticdrive, and/or other external memory.

FIG. 11C illustrates one or more implementations of shared busneuromorphic computerized system 1145 comprising micro-blocks 1140,described with respect to FIG. 1113, supra. The system 1145 of FIG. 11Cmay utilize shared bus 1147, 1149 to interconnect micro-blocks 1140 withone another.

FIG. 11D illustrates one implementation of cell-based neuromorphiccomputerized system architecture configured to implement activity-basedplasticity mechanism in a spiking network is described in detail. Theneuromorphic system 1150 may comprise a hierarchy of processing blocks(cells blocks). In some implementations, the lowest level L1 cell 1152of the apparatus 1150 may comprise logic and memory blocks. The lowestlevel L1 cell 1152 of the apparatus 1150 may be configured similar tothe micro block 1140 of the apparatus shown in FIG. 1113. A number ofcell blocks may be arranged in a cluster and may communicate with oneanother via local interconnects 1162, 1164. Individual clusters may formhigher level cell, e.g., cell L2, denoted as 1154 in FIG. 11d .Similarly, several L2 clusters may communicate with one another via asecond level interconnect 1166 and form a super-cluster L3, denoted as1156 in FIG. 11D. The super-clusters 1154 may communicate via a thirdlevel interconnect 1168 and may form a next level cluster. It will beappreciated by those skilled in the arts that the hierarchical structureof the apparatus 1150, comprising four cells-per-level, is merely oneexemplary implementation, and other implementations may comprise more orfewer cells per level, and/or fewer or more levels.

Different cell levels (e.g., L1, L2, L3) of the apparatus 1150 may beconfigured to perform functionality various levels of complexity. Insome implementations, individual L1 cells may process in paralleldifferent portions of the visual input (e.g., encode individual pixelblocks, and/or encode motion signal), with the L2, L3 cells performingprogressively higher level functionality (e.g., object detection).Individual ones of L2, L3, cells may perform different aspects ofoperating a robot with one or more L2/L3 cells processing visual datafrom a camera, and other L2/L3 cells operating motor control block forimplementing lens motion what tracking an object or performing lensstabilization functions.

The neuromorphic apparatus 1150 may receive input (e.g., visual input)via the interface 1160. In one or more implementations that may beapplicable to, for example, interfacing with computerized spikingretina, or image array, the apparatus 1150 may provide feedbackinformation via the interface 1160 to facilitate encoding of the inputsignal.

The neuromorphic apparatus 1150 may provide output via the interface1170. The output may include one or more of an indication of recognizedobject or a feature, a motor command, a command to zoom/pan the imagearray, and/or other outputs. In some implementations, the apparatus 1150may perform all of the I/O functionality using single I/O block (notshown).

The apparatus 1150, in one or more implementations, may interface toexternal fast response memory (e.g., RAM) via a high bandwidth memoryinterface (not shown), thereby enabling storage of intermediate networkoperational parameters (e.g., spike timing, neuron state, and/or otherparameters). In one or more implementations, the apparatus 1150 mayinterface to external memory via a lower bandwidth memory interface (notshown) to facilitate program loading, operational mode changes,retargeting, and/or other operations. Network node and connectioninformation for a current task may be saved for future use and flushed.Previously stored network configuration may be loaded in place of thenetwork node and connection information for the current task, asdescribed for example in co-pending and co-owned U.S. patent applicationSer. No. 13/487,576, entitled “DYNAMICALLY RECONFIGURABLE STOCHASTICLEARNING APPARATUS AND METHODS”, incorporated supra.

In one or more implementations, one or more portions of the apparatus1150 may be configured to operate one or more learning rules, asdescribed for example in owned U.S. patent application Ser. No.13/487,576 entitled “DYNAMICALLY RECONFIGURABLE STOCHASTIC LEARNINGAPPARATUS AND METHODS”, filed Jun. 4, 2012, incorporated herein byreference in its entirety. In one such implementation, one block (e.g.,the L3 block 1156) may be used to process input received via theinterface 1160 and to provide a reinforcement signal to another block(e.g., the L2 block 1156) via interval interconnects 1166, 1168.

In one or more implementations, networks of the apparatus 1130, 1145,1150 may be implemented using Elementary Network Description (END)language, described for example in U.S. patent application Ser. No.13/239,123, entitled “ELEMENTARY NETWORK DESCRIPTION FOR NEUROMORPHICSYSTEMS”, filed Sep. 21, 2011, and/or High Level NeuromorphicDescription (HLND) framework, described for example in U.S. patentapplication Ser. No. 13/385,938, entitled “TAG-BASED APPARATUS ANDMETHODS FOR NEURAL NETWORKS”, filed Mar. 15, 2012, each of the foregoingincorporated supra. The HLND framework may be augmented to handle anevent based update methodology, for example, such as that described inU.S. patent application Ser. No. 13/588,774, entitled “APPARATUS ANDMETHODS FOR IMPLEMENTING EVENT-BASED UPDATES IN SPIKING NEURON NETWORK”,filed Aug. 17, 2012, the foregoing being incorporated herein byreference in its entirety. The networks may be updated using anefficient network update methodology, described, for example, U.S.patent application Ser. No. 13/239,259, entitled “APPARATUS AND METHODFOR PARTIAL EVALUATION OF SYNAPTIC UPDATES BASED ON SYSTEM EVENTS”,filed Sep. 21, 2011 and/or U.S. patent application Ser. No. 13/385,938,entitled “APPARATUS AND METHODS FOR EFFICIENT UPDATES SPIKING NEURONNETWORKS”, filed Jul. 27, 2012, each of the foregoing being incorporatedherein by reference in its entirety.

In some implementations, the END may be used to describe and/or simulatelarge-scale neuronal model using software and/or hardware engines. TheEND may allow for optimal architecture realizations comprising ahigh-performance parallel processing of spiking networks withspike-timing dependent plasticity. Neuronal network configured inaccordance with the END may comprise units and doublets, the doubletsbeing connected to a pair of units. Execution of unit update rules forthe plurality of units is order-independent and execution of doubletevent rules for the plurality of doublets is order-independent.

In one or more implementations, the efficient update methodology (e.g.,for adjusting input connections and/or inhibitory traces) may compriseperforming of pre-synaptic updates first, followed by the post-synapticupdates, thus ensuring the up-to-date status of synaptic connections.

The activity based plasticity described herein may advantageouslyprovide a mechanism for retaining strong but temporarily inactivesynapses via anti-causal potentiation. In some realizations, suchsynapses are retained irrespective of the activity of post synapticneuron. Such efficacy retention mechanism may allow for long intervals(e.g., from several seconds to several minutes) between slow featuresthat may be processed (pulled together) by the neuron. The term slowfeature analysis may be used to describe features and aspects of theinput that persist between consecutive presentations (frames) of theinput. In one or more implementations, an input within the input framemay be regarded as a slow feature if it is detectable (with a sufficientaccuracy, e.g., 75%) for many (e.g., 5-5,000 frames for a 25 fps framerate) frames. By way of an example, for an input I containing typicalvideo recording at 25 fps of a street and/or a room with people walkingaround, a function of the input f(I) may return the ‘TRUE’ value forinput frames comprising representations of a human face; the functionf(I) may return ‘FALSE’ for frames that do not contain a face. In someapproaches, logical functions that, over a broad range of inputs (e.g.video input), may persistently provide either TRUE or FALSE responsewith relatively little variability (e.g., less than 10% of the frames,in some implementations when processing a typical ensemble of framescontaining natural images) may be regarded as slow features.

The selectivity of the neurons receptive field is obtained by adifferent mechanism (the invention) where the depression is related topost-synaptic activity but only applied in the event of a presynapticspike triggering a synapse.

Various aspects of the disclosure may advantageously be applied to,inter alia, the design and operation of large spiking neural networksconfigured to process streams of input stimuli, in order to aid indetection and functional binding related to an aspect of the input.

In some implementations, the activity-based and/or plasticity modulationmechanisms described herein may be implemented in a spiking neuron of anetwork, or in a connection of the network.

It is appreciated by those skilled in the arts that above implementationare exemplary, and the framework of the disclosure is equally compatibleand applicable to processing of other information. For example, theframework may be applied in information classification using a database,where the detection of a particular pattern can be identified as adiscrete signal similar to a spike, and where coincident detection ofother patterns influences detection of a particular one pattern based ona history of previous detections in a way similar to an operation ofexemplary spiking neural network.

Advantageously, exemplary implementations of the various aspects of thepresent innovation are useful in a variety of devices including withoutlimitation prosthetic devices, autonomous and robotic apparatus, andother electromechanical devices requiring sensory processingfunctionality. Examples of such robotic devices may includemanufacturing robots (e.g., automotive), military, medical (e.g.processing of microscopy, x-ray, ultrasonography, tomography). Examplesof autonomous vehicles include rovers, unmanned air vehicles, underwatervehicles, smart appliances (e.g. ROOMBA®), Lego® robotic toys, and/orother devices.

Implementations of the principles of the disclosure may be applicable tovideo data compression and processing in a wide variety of stationaryand portable devices (e.g., for example, smart phones, portablecommunication devices, notebook, netbook and tablet computers,surveillance camera systems, and practically any other computerizeddevice configured to process vision data).

Implementations of the principles of the disclosure are furtherapplicable to a wide assortment of applications including computer humaninteraction (e.g., recognition of gestures, voice, posture, face, and/orother aspects), controlling processes (e.g., an industrial robot,autonomous and other vehicles), augmented reality applications,organization of information (e.g., for indexing databases of images andimage sequences), access control (e.g., opening a door based on agesture, opening an access way based on detection of an authorizedperson), detecting events (e.g., for visual surveillance or people oranimal counting, tracking), data input, financial transactions (paymentprocessing based on recognition of a person or a special payment symbol)and many others.

Advantageously, the disclosure can be used to simplify tasks related tomotion estimation, such as where an image sequence is processed toproduce an estimate of the object position (and hence velocity) eitherat individual points in the image or in the 3D scene, or even ofposition of the camera that produces the images. Examples of such tasksmay include: ego-motion, i.e., determining the three-dimensional rigidmotion (rotation and translation) of the camera from an image sequenceproduced by the camera; following the movements of a set of interestpoints or objects (e.g., vehicles or humans) in the image sequence andwith respect to the image plane.

In another approach, portions of the object recognition system may beembodied in a remote server, comprising a computer readable apparatusstoring computer executable instructions configured to perform patternrecognition in data streams for various applications, such asscientific, geophysical exploration, surveillance, navigation, datamining (e.g., content-based image retrieval). A myriad of otherapplications exist that will be recognized by those of ordinary skillgiven the present disclosure.

Although the system(s) and/or method(s) of this disclosure have beendescribed in detail for the purpose of illustration based on what iscurrently considered to be the most practical and preferredimplementations, it is to be understood that such detail is solely forthat purpose and that the disclosure is not limited to the disclosedimplementations, but, on the contrary, is intended to covermodifications and equivalent arrangements. For example, it is to beunderstood that the present disclosure contemplates that, to the extentpossible, one or more features of any implementation can be combinedwith one or more features of any other implementation. Moreover, givensteps of a method may be added or subtracted, or their order ofperformance permuted.

What is claimed:
 1. An apparatus, comprising: a memory; and at least oneprocessor coupled to the memory, the at least one processor configured,based on a response by a neuron: to increase an efficacy of a connectionconfigured to provide input to the neuron prior to the response; and toadjust the efficacy of the connection configured to provide input to theneuron subsequent to the response; wherein: the adjustment of theefficacy is determined based on a rate of the response; the responsecomprises one or more spikes generated by the neuron based on the input;the rate of the response comprises an average rate determined based onat least a number of the one or more spikes occurring within an intervalpreceding the response; the increasing of the efficacy is characterizedby a first rule; and the adjustment of the efficacy comprises increasingthe efficacy based on a second rule, a magnitude of the increase beingdetermined based at least on the average rate being below a threshold.2. The apparatus of claim 1 wherein: the input is characterized by arefresh period of 40 ms; and the rate of the response is based on anaverage rate of responses by the neuron determined within a time windowselected from the range between 1 s and 2000 s, inclusive.
 3. Theapparatus of claim 1 wherein the adjustment of the efficacy furthercomprises a decrease in the efficacy, a magnitude of the decrease beingbased on the average rate being above the threshold.
 4. The apparatus ofclaim 3 wherein the magnitude of the decrease and the magnitude of theincrease are each independent of a time interval between the responseand the input.
 5. The apparatus of claim 4 wherein the magnitude of theincrease is configured to be smaller than the magnitude of the decrease,and the magnitude of the increase is characterized by the first rule. 6.The apparatus of claim 4, wherein: the input is characterized by arefresh rate of 25 Hz; the threshold is selected from the range between0.05 Hz and 0.2 Hz; and a ratio of the magnitude of the increase basedon the second rule to the magnitude of the decrease is selected from therange between 0.05 and 0.2 inclusive.
 7. The apparatus of claim 1wherein the adjustment is based at least in part on a time intervalbetween the response and another response preceding the response.
 8. Theapparatus of claim 1 wherein: the efficacy increase comprises increasingthe efficacy based on a delay between the response and the input beingbelow the threshold; and the efficacy is maintained based on the delaybetween the response and the input being above the threshold.
 9. Theapparatus of claim 8, wherein: the input is characterized by a refreshrate; the refresh rate comprises 40 ms; and the threshold is selectedfrom the range between 500 ms and 1000 ms, inclusive.
 10. The apparatusof claim 1 wherein: the input is characterized by a refresh rate of 40ms; and the efficacy increase comprises increasing the efficacy whendelay between the response and the input is below the threshold, thethreshold is selected from the range between 20 ms and 200 ms,inclusive.
 11. A sensory processing spiking neuron network apparatus,comprising: a connection configured to provide an input to a neuronconfigured to generate a response based on the input; wherein theconnection is further configured to be: potentiated when the input iswithin an interval from the response; adjusted when the input occurssubsequent to the response, the adjustment being determined based onactivity of the neuron prior to the response.
 12. The apparatus of claim11, wherein: the connection is characterized by a connection efficacyconfigured to delay or advance a response generation by the neuron; thepotentiation comprises increase of the efficacy configured to advancegeneration of another response subsequent to the response; if theresponse occurs after a second time interval after the input, theefficacy is maintained at a level corresponding to the efficacy prior tothe response; and the second time interval comprises at least a portionof the interval.
 13. A non-transitory computer-readable medium havingencoded thereon program code, the program code executed by a processorand comprising: program code to update a connection configured toprovide stimulus to an artificial neuron program code to potentiate theconnection if the stimulus precedes a response generated by theartificial neuron; program code to potentiate the connection if theresponse precedes the stimulus and a neuron activity is below athreshold level; and ;and program code to depress the connection whenthe response precedes the stimulus and the neuron activity is above thethreshold level.
 14. The non-transitory computer-readable medium ofclaim 13, wherein: the depression of the connection is configured todelay another response by the artificial neuron subsequent to theresponse; and the potentiation of the connection is configured toadvance the another response.
 15. The non-transitory computer-readablemedium of claim 14, wherein: the connection is characterized by alatency configured to delay provision of the stimulus to the artificialneuron; the connection potentiation comprises a reduction in thelatency; and the connection depression comprises an increase in thelatency.
 16. The non-transitory computer-readable medium of claim 15,wherein: the connection is characterized by a weight configured toaffect generation of the response by the artificial neuron such that alarger weight corresponds to a greater probability of the response beinggenerated; the connection potentiation comprises an increase in theweight; and the connection depression comprises a reduction in theweight.
 17. A method of managing a connection in a neuron network basedon at least one signal from a neuron, the method comprising: receivingat least one input via the connection; sending the at least one signalat a time proximate to the received at least one input; if the at leastone signal is sent prior to the reception of the input, demoting theconnection; and if the at least one signal is sent after the receptionof the input, promoting the connection.
 18. The method of claim 17,wherein: the demotion of the connection decreases a probabilityassociated with signal generation at the neuron; and the promotion ofthe connection increases a probability associated with signal generationat the neuron.
 19. The method of claim 17, wherein: the demotion of theconnection is configured to delay signal generation at the neuron; andthe promotion of the connection is configured to advance signalgeneration at the neuron.
 20. The method of claim 17, wherein the timeproximate to the received at least one input comprises an intervalduring which the at least one input is received.
 21. The method of claim20, wherein a neuron response is governed by a minimum feature timescale, the interval being greater than the minimum feature time scale.22. An apparatus comprising means for increasing an efficacy of aconnection configured to provide input to a neuron prior to a responseby the neuron; and means for adjusting the efficacy of the connectionconfigured to provide input to the neuron subsequent to the response;wherein: the adjustment of the efficacy is determined based on a rate ofthe response; the response comprises one or more spikes generated by theneuron based on the input; the rate of the response comprises an averagerate determined based on at least a number of the one or more spikesoccurring within an interval preceding the response; the increasing ofthe efficacy is characterized by a first rule; and the adjustment of theefficacy comprises increasing the efficacy based on a second rule, amagnitude of the increase being determined based at least on the averagerate being below a threshold.
 23. The apparatus of claim 22, wherein theadjustment of the efficacy further comprises a decrease in the efficacy,a magnitude of the decrease being based on the average rate being abovethe threshold.
 24. The apparatus of claim 23, wherein the magnitude ofthe decrease and the magnitude of the increase are each independent of atime interval between the response and the input.
 25. An apparatuscomprising means for updating a connection configured to providestimulus to an artificial neuron; means for potentiating the connectionif the stimulus precedes a first response generated by the artificialneuron; means for potentiating the connection if the first responseprecedes the stimulus and a neuron activity is below a threshold level;and means for depressing the connection if the first response precedesthe stimulus and the neuron activity is above the threshold level. 26.The apparatus of claim 25, wherein: the depression of the connection isconfigured to delay a second response by the artificial neuronsubsequent to the first response; and the potentiation of the connectionis configured to advance the second response.
 27. The apparatus of claim26, wherein: the connection is characterized by a latency configured todelay provision of the stimulus to the artificial neuron; the connectionpotentiation comprises a reduction in the latency; and the connectiondepression comprises an increase in the latency.
 28. An apparatuscomprising: means for managing a connection in a neuron network based onat least one signal from a neuron; means for receiving at least oneinput via the connection; means for sending the at least one signal at atime proximate to the received at least one input; means for demotingthe connection if the at least one signal is sent prior to the receptionof the input; and means for promoting the connection if the at least onesignal is sent after the reception of the input.
 29. The apparatus ofclaim 28, wherein: the demotion of the connection decreases aprobability associated with signal generation at the neuron; and thepromotion of the connection increases a probability associated withsignal generation at the neuron.
 30. The apparatus of claim 28, wherein:the demotion of the connection is configured to delay signal generationat the neuron; and the promotion of the connection is configured toadvance signal generation at the neuron.